Serial Attached SCSI (Small Computer System Interface), or SAS, is a connection-oriented protocol that allows storage devices, like servers and disk drives, to communicate through a network of high-speed serial physical interconnects. Connections between a host device and a target drive are managed by intermediate devices called expanders. SAS expanders act as connection management agents, much like a switch element, having physical connections to multiple host devices or disk drives simultaneously.
The SAS specification defines three transport level protocols, which are used in a SAS topology under different circumstances: Serial SCSI Protocol (SSP), Serial Management Protocol (SMP), Serial Advanced Technology Attachment Tunnelling Protocol (STP). An STP connection is set up when a SAS host device accesses a Serial Advanced Technology Attachment (SATA) disk drive or vice versa. Once the connection is set up, data transfer occurs between the host device and the disk as per the SATA protocol, at a half-duplex rate.
The SATA protocol makes use of 32-bit constructs called Dwords and special control Dwords, called primitives. All SATA primitives are either repeated or singular. Repeated primitives are redundant control Dwords that are required to be sent two at a time. Single primitives are control Dwords with no such restriction. A repeated primitive indicates a data transmitter or receiver state whereas a single primitive is associated instead with a particular Dword position. For example, an X_RDY repeated primitive indicates that a data frame (referred to as a frame information structure, or FIS) is ready for transmission. Likewise, an R_RDY repeated primitive indicates that a data receiver is ready for frame reception. These primitives indicate a transmitter or receiver state. In contrast, a single primitive such as SOF marks a specific Dword (in this case, the position of the start of frame). For reference, all SATA primitives, along with their respective functions, are listed in the table below:
TABLE 1DMATDMA terminate. Sent by a data receiver inSingleorder to terminate a FIS transfer.EOFEnd of frame Dword. If a data receiver is inSinglethe midst of receiving a FIS, the preceding dataDword shall be considered as the CRC Dwordof the FIS.HOLDHold data transmission. Sent by a dataRepeatedtransmitter when it temporarily has no moredata to send. Also sent by a data receiver tothrottle the flow of data in the forward channel.HOLDAHold acknowledge. Sent by a data transmitterRepeatedor receiver to acknowledge reception of aHOLD primitive.PMACKPower management acknowledge. Sent by aSingledisk to acknowledge a power managementrequest.PMNAKPower management denial. Sent by a disk toSinglerefuse a power management request.PMREQ_PPower management request to partial. Sent by aRepeatedhost device to place a disk into a powermanagement partial mode.PMREQ_SPower management request to slumber. Sent byRepeateda host device to place a disk into a powermanagement slumber mode.R_ERRReception error. Sent by a data receiver toRepeatedindicate that a FIS was not was receivedcorrectly.R_IPReception in progress. Sent by a data receiverRepeatedto indicate that FIS reception is in progress.R_OKReception with no error. Sent by a data receiverRepeatedindicating that a FIS was received correctly.R_RDYReceiver ready. Sent by a data receiver toRepeatedindicate that it is ready to receive a FIS.SOFStart of frame. Precedes first data Dword ofSingleFIS.SYNCSynchronization. Indicates an idle state.RepeatedWTRMWait for frame termination. Sent by a dataRepeatedtransmitter to indicate that a FIS has beentransmitted and is now awaiting an R_OK orR_ERR from the data receiver.X_RDYTransmission data ready. Sent by dataRepeatedtransmitter to initiate a FIS transfer.
A typical FIS transfer via the SATA protocol is shown in FIG. 1. Note that an end device, whether host or disk, can act as either a transmitter or a receiver.
After an STP connection is set up, a SAS expander acts, by simple analogy, as nothing more than a wire would between the initiator and target. In fact, under normal operation, connection teardown can only be initiated by an end device. However, after an expander has set up an STP connection between a SAS host device and a SATA disk, the expander must still terminate flow control on a hop-by-hop basis. This is because the SATA protocol has defined a flow control mechanism in which forward data flow must be stopped within 21 Dwords of a data receiver generating HOLD primitives. Because of the latency introduced by the expander into the data path, flow control must be terminated at each hop (at each expander port), rather than by relying on an end device's link layer to do so. An example FIS transfer between two end devices with flow control on the back channel is shown in FIG. 2.
Another FIS transfer example featuring flow control is shown in FIG. 3. In this example, the data transmitter generates HOLD primitives after running out of data to send in the middle of a FIS. Here, unlike the backward channel flow control case, it is not catastrophic if HOLDA primitives are received after 21 Dwords, even though this is a clear violation of the SATA protocol. This is because the data transmitter does not take any consequential action after receiving either HOLDA primitives or R_IP primitives.
To properly terminate SATA flow control on a hop-by-hop basis, an expander port must respond to HOLD primitives with HOLDA primitives quickly enough that the total loop time, measured from the initiator of the HOLD primitives, meets the 21-Dword requirement. An elastic first-in-first-out queue (FIFO) for buffering a stream of data between sender and receiver is also required in each connection data path, both to account for this loop time delay when an expander port initiates flow control and to allow the synchronization of data from one expander port clock domain to another. When an expander port is acting as a data receiver, it should generate HOLD primitives (initiate flow control) once its forward channel FIFO does not have enough space to accommodate at least one loop time's worth of data. Ideally, this ensures that the FIFO will never overflow. FIG. 4 is a basic block diagram of an expander while inside a connection. Note that only the in-connection expander ports are shown here.
Termination of STP flow control in a SAS expander preferably provides a forward data channel FIFO per port and control logic to ensure that the following desirable characteristics, or requirements, are met (as stated in the SAS specification as published by the Internabonal Committee for Information Technology Standards of the American National Standards Institute as ANSI INCITS 376-2003):
1. When an STP port (i.e. an expander port in an STP connection) is transmitting a frame and receives HOLD, it shall transmit no more than 20 data Dwords for the frame and respond with HOLDA.
2. When an STP port is receiving a frame and its buffer begins to fill up, it shall transmit HOLD. After transmitting HOLD, it shall accept the following number of data Dwords for the frame:
a) 24 Dwords when the line rate is 1.5 Gbps; or
b) 28 Dwords when the line rate is 3.0 Gbps
3. When a SATA host port in an STP/SATA bridge (i.e. an expander port connected to a SATA disk) is receiving a frame from a SATA physical link, it shall transmit a HOLD primitive when it is only capable of receiving 21 more Dwords.
4. When a SATA host port in an STP/SATA bridge is transmitting a frame to a SATA physical link, it shall transmit no more than 19 data Dwords after receiving HOLD.
5. The data receiver (backward channel) must send HOLD primitives (initiate flow control) only while it is in the middle of receiving a FIS or after receiving a FIS but before R_OK/R_ERR is transmitted.
6. The data transmitter (forward channel) must send HOLD primitives (initiate flow control) only while it is in the middle of transmitting a FIS.
Note that each characteristic/requirement is specific to either the forward channel or the backward channel. This implies that the expander port control logic should be aware whether it is acting as a data transmitter or receiver (i.e., whether framed data is received on the ingress phy or is outgoing on the egress phy where “phy” is defined as the physical layer that provides the electrical and mechanical interface required for transmission and reception of data packets transferred across the given medium—e.g., serial bus). Each port should also be cognizant of FIS boundaries as HOLDA primitives are sent as a response to received HOLD primitives only while in the middle of a frame transfer. Furthermore, since a SAS topology allows for daisy-chained expanders, in which case a STP connection can straddle more than one expander device, the STP flow control strategy should satisfy these requirements both when an expander port is connected to an end device and when it is connected to another expander.
FIG. 5 illustrates a known example of how, based on the aforementioned requirements, flow control initiated by a data receiver can be propagated back to the data transmitter and terminated at each hop, while in a multi-expander STP connection.
While an expander port could conceivably ensure the flow control loop time requirement is met while maintaining data Dword integrity in connection by incorporating a complete SATA link state machine per port, this approach would be very costly. A less costly approach would therefore be desirable. There are very few approaches that specifically address the problem of STP flow control in SAS expanders. Several approaches to flow control, although in entirely different contexts, include U.S. Pat. No. 5,696,990 issued Dec. 9, 1997 to Rosenthal et al., U.S. Pat. No. 6,249,756 issued Jun. 19, 2001 to Bunton et al., and U.S. Pat. No. 5,237,660 issued Aug. 17, 1993 to Weber et al., each of which is incorporated herein by reference in their entirety.
It is, therefore, desirable to provide an improved STP flow control strategy to address the aforementioned flow control problem in a gate-efficient manner.